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CROSS-REFERENCES TO RELATED APPLICATIONS 

The instant application is a continuation of Applicant's prior application 08/485,384 (to 
issue as patent 6,025,882) and a continuation-in-part of applicant's prior applications 08/483,206 
(issued as patent 5,978,035) 08/483,210 (issued as patent 5,986,707) 08/485,382, 08/485,383, 
08/485,385 and 08/488,222 all filed June 7, 1995. 

All of these documents are hereby incorporated in their entirety by reference. The reader 
is directed to those applications, and the references made therein, for additional and background 
information. 



FIELD OF INVENTION 

Generally, the domain where the instant invention can be applied is any process, system, 
device, or product made by a process, which involves a processing element whereby a first 
information bearing signal (often, a three-dimensional data construct consisting of a two- 
dimensional image plane and a third dimension of time — e.g., a television or motion picture 
image stream) is transformed into a second information bearing signal of similar type. Further, 
these signals, though of similar type, are, generally, different in some characteristic, such as 
resolution or definition. Examples of such areas of potential application include, without 
limitation, devices or processes that: encode a higher-definition television or video signal into 
a lower-definition television or video signal suitable for transmission, display or recording; 
record, transmit, decode or display such an encoded signal; transduce or transfer an image 
stream from an imaging element to a transmission or storage element, such as a television 
camera or film chain; transfer an image stream from a signal input to a recording medium, such 
as a videotape or videodisc recorder; transfer an image stream from a recording medium to a 
display element, such as a videotape or videodisc player; transfer data representing images from 
a computer memory element to a display element, such as a framestore or frame buffer; 
synthesize an image output stream from a mathematical model, such as a computer graphic 
rendering component; modify or combine image streams, such as image processing components, 
time-base correctors, signal processing components, or special effects components; products that 
result from the foregoing; and many other devices, processes and products that fall within the 
realm of motion picture and television engineering. 

The invention, rather than comprising any of the foregoing, constitutes a method (e.g., 
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a process or algorithm) for selecting and arranging information derived from the first 
information bearing signal, which processed information will then be used to create a second 
information bearing signal. Thus, the inventive matter comprises algorithms for selecting and 
arranging information; as well as otherwise standard devices or processes that incorporate those 
algorithms, and specially modified or constructed devices and processes that incorporate those 
algorithms. In particular, methods and devices for incorporating additional information, such as 
HDTV side strips, into the blanking intervals of a previous frame, were developed by Inventor 
to affect the foregoing, but this can also be independently practiced to great advantage. 

In particular, the parent method comprises a class of techniques for doing so in a time- 
varying manner such that the artifacts introduced by the process or device involved become less 
noticeable to a human viewer than if standard techniques or algorithms are applied. It is this 
"greater than standard" variability over time, of the selection of information derived from the 
source signal and/or its arrangement into the information which will comprise the destination 
signal, that results in the improvement; and, in order to achieve an improvement, the variation 
over time must be of sufficient magnitude. 

To distinguish patterns, techniques, processes or algorithms with variability sufficient to 
achieve a reduction in perceptual artifacts, from those which do not reduce (and may even 
increase) such artifacts, the words complex or complexly will be occasionally used herein to 
modify terms such as sampling pattern, time variation or time-varying STS (Spatial Temporal 
Signature, described below); and the words complex or complexity (sometimes used with 
variation) will be occasionally used to point out inventive techniques which can be used to 
achieve a sufficient magnitude of variation for effective practice. 

Examples of "complex variable STS" or "complexity" when used to describe a variable 
STS or sampling pattern herein comprise, singly or in combination: rectangular (MxN) multi- 
pixel cells with a sampling scheme that decimates information significantly less than by a factor 
of MxN for a particular frame; multi-pixel cells that are non-rectangular; multi-pixel cells where 
more than a single pixel is sampled for a particular frame; pseudo-random and/or non-contiguous 
multi-pixel cells; distributions of multi-pixel cells of non-uniform size or shape over an image, 
or from frame to frame; multiplication weights for information points not equal to 1 (or a 
constant), or which change over a cell or frame, or from frame to frame; sampling schemes that 
utilize non-uniform time slices from frame to frame or over a frame; or as otherwise described 
herein. Thus, the word "complex", unless otherwise indicated, as used herein means intricate 
or sophisticated, and does not refer to numbers comprising real and imaginary parts. 
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BACKGROUND ART 

The intended practitioner of the present invention is someone who is skilled in designing, 
5 implementing, integrating, building, creating, programming or utilizing processes, devices, 
systems and products, such as described in the first paragraph of this disclosure. That is, one 
skilled in the art required to practice the instant invention is capable of one or more of the 
following: design and/or construction of devices, systems, hardware and software (i.e., 
programming) for motion picture and television production, motion picture and television post 
10 production, signal processing, image processing, computer graphics, and the like. That is, 
motion picture and television engineers, computer graphic system designers and programmers, 
image processing system designers and programmers, digital software and hardware engineers, 
P communication and information processing engineers, applied mathematicians, etc. 
ill Those skilled in the art know how to accomplish such tasks as to: design and construct 

Sfe devices, design and integrate systems, design software for and program those devices and 
j|i systems, and utilize those devices and systems to create information product, which devices and 
p systems transfer and/or transform information derived from image streams. Such devices and 

j H 

m systems comprise well known digital or analog electronic hardware, and software, components, 
p The details of accomplishing such standard tasks are well known and within the ken of those 
20 skilled in these arts; are not (in and of themselves) within the scope of the instant invention; and, 
|jl will be referred to but not described in detail in the instant disclosure. 1 
H Rather, what will be disclosed are novel algorithms for selecting and arranging the 

w information transferred or transformed by those systems, devices and processes; or the selection 
and arrangement of information comprising various information products such as magnetic or 
25 optical media containing, or transmissions of, information bearing signals; as well as methods 
and devices for incorporating additional information, such as HDTV side strips, into the 
blanking intervals of a previous frame, which may be used to affect the foregoing or 
independently to great advantage. Disclosed will be what one skilled in the art will need to 
know, beyond that with which he is already familiar, in order to implement the instant invention. 
30 These algorithms and system designs will be presented by description, algebraic formulae and 
graphically, indicating how information data points or "pixels" are to be selected from data 
derived from a first information signal, and arranged, manipulated and/or combined to create 
the pixels or data points that will then comprise a second information signal; as is standard and 
frequent practice in the field of motion picture and television engineering. 2 
35 These descriptions, formulae and illustrations are such as to completely and clearly 
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specify algorithms which may be implemented in a straightforward manner by programming a 

programmable computer imaging device such as a frame buffer. 

For example, the programmable frame buffers (some with onboard special-purpose 

microprocessors for graphics and/or signal processing) suitable for use with personal computers, 
5 workstations or other digital computers, along with off-the-shelf assemblers, compilers, 

subroutine libraries, or utilities, routinely provide as standard features, capabilities which permit 

a user to (among other tasks): digitize a frame of a video signal in many different formats 

including higher-than-tele vision resolutions, standard television resolutions, and lower-than- 

television resolutions, and at 8- 16- 24- and 32-bits per pixel; display a video signal in any of 
10 those same formats; change, under program control, the resolution and/or bit-depth of the 

digitized or displayed frame; transfer information between any of a) visible framestore memory, 

b) blind (non-displayed) framestore memory, and c) host computer memory, and d) mass storage 
P- (e.g. , magnetic disk) memory, on a pixel-by-pixel, line-by-line, or rectangle-by-rectangle basis. 3 
pi Thus, off-the-shelf devices provide the end user with the ability to: digitize high- 

ly resolution video frames; access the individual pixels of those frames; manipulate the information 

from those pixels under generalized host computer control and processing, to create arbitrarily 
0 processed pixels; and, display lower-resolution frames, suitable for recording, comprising those 
^ processed pixels. These off-the-shelf capabilities are sufficient to implement an image processing 
p system embodying the information manipulation algorithms or system designs specified herein. 

Similarly, higher performance and throughput (as well as higher cost and more 
In programming effort), programmable devices, suitable for broadcast or theatrical production 
B tasks, provide similar and much more sophisticated capabilities, including micro-coding whereby 
*** image processing algorithms can be incorporated into general purpose hardware, are available 

as off-the-shelf programmable systems. 4 
25 Additionally, specialized (graphic and image processing) programmable microprocessors 

are available for incorporation into digital hardware capable of providing special-purpose or 

general-purpose (user-programmable) image manipulation functions. 5 

Further, it is well known by those skilled in the art how to adapt processes that have been 

implemented as software running on programmable hardware devices, to designs for special 
30 purpose hardware, which may then provide advantages in cost vs. performance. 

In summary, the disclosure of the instant invention will focus on what is new and novel 

and will not repeat the details of what is known in the art. 

Additionally, in the disclosure, a number of specific processes, devices or systems into 

which the algorithms of the instant invention can be incorporated, and applications to which the 
35 algorithms of the instant invention can be applied, will be pointed out. The intended practitioner 
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of the instant invention is already skilled in the art of: building and/ or programming some one 
(or more) of these devices; integrating and/or programming for some one (or more) of these 
systems; implementing (and/or utilizing to create information product) some one (or more) of 
these application systems; or, designing processes and devices to manipulate information that has 
5 been modified by random noise or some specified process in order to counteract that 
manipulation to some degree, including encoding, decoding, reconstruction, compression and 
decompression. 

As such, the intended practitioner has within their skill in the art the knowledge to design 
and create and/or use such devices, processes or systems which "push pixels around" from some 
10 first or source information bearing signal, in a specified manner, to create a second or 
destination information bearing signal. 

Examples include, without limitation: special purpose hardware or programmable devices 
that permit the implementation of specific or arbitrary image processing functions (for example, 
W image filtering, blurring, sharpening, warping, zooming, sizing, tilting, compositing, wiping, 
|1> keying, matting, texture mapping, colorization 6 , posterization, etc.); and, hardware devices that 
Bt transfer or transform information, including image streams, from a source form to a destination 
S*iJ form (encoders, decoders, cameras, recorders, players, displays, switchers, timebase correctors, 
s noise reducers, compressors, decompressors, modulators, demodulators, etc.). 

'% t The intended practitioner is already in command of the knowledge required to design and 

go build such devices and application systems that select, arrange and manipulate the pixels of 
image streams, in various previously specified manners. What is provided herein is the 
S specification for a novel manner for selection, arrangement and manipulation of the pixels 
comprising such image streams. What is claimed as inventive are these algorithms and/or the 
incorporation of the novel algorithms into an otherwise standard device. 
25 For example, Inventor's own film colorization system (US Patent Number 4,606,625) 

discloses a system comprising all the elements needed to input an image stream, process it by 
applying colorization and other digital filtering algorithms, and output a processed image stream. 
Replacing the colorization process or algorithm element of that prior invention with some one 
of the algorithms herein would result in a system for practicing the encoding envisioned herein 
30 on an information bearing signal derived from motion picture film or video. 

Additionally, in the disclosure, it will be pointed out that a number of the parameters 
(e.g., the set five weights Ka, Kb, Kc, Kd and Kx for the 4+1:1 pattern) can be varied, and 
may be adjusted or "fine tuned". It is common practice for users of television imaging systems 
to adjust or "fine tune" operating parameters via physical controls or data entry. For example, 
35 with a computer image processing capability (such as those provided with computer paint 
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systems or special effects systems — e.g., digital filters that blur, sharpen, colorize, posterize, 
size, stretch, warp or otherwise manipulate an image or image stream) it is common practice for 
a practitioner to adjust various parameters that control the processing results via numeric 
keyboard input, or by physical or graphic controls. Considerable time may be spent making 
5 adjustments to the process, for any particular image input to the system, to achieve the desired 
results. The determination of what constitutes such desired results, and when those goals have 
been adequately approximated are, in the motion picture and television industry, most often a 
matter of subjective artistic taste; what one practitioner (or, more importantly the creative 
director to whom he reports) may deem successful, might be entirely unacceptable to another 

10 team. The specific adjustments made, will vary from case to case and will depend upon both the 
characteristics of the source material and the results hoped for after processing. How the system 
parameters should be adjusted is often not known ahead of time. Parameters are adjusted until 

jj a desired result is achieved or approximated. 

W Further, some image processing systems, such as Inventor's own film colorization system 

|l (US Patent Number 4,606,625) permit the application of algorithms with separately adjusted 
P parameters (such as color specification or digital filtering parameters) to be applied to distinct 
7^ areas of images in an image stream. Typically (in Inventor's systems) parameters are specified 
s and adjusted in a small percentage of images from a moving sequence; the computer being 

^ capable of interpolating the practitioner's instructions for the intervening frames. Thus, it is not 
20 only common practice to fine tune parameters for a particular image, but also for individual 
111 areas of a given image. 

K Further still, it is common practice to "color correct" each individual scene of a motion 

picture film as it is printed, or transferred to video. Considerable time and effort, by highly 
skilled practitioners, is spent adjusting the parameters separately for individual scenes to achieve 

25 a desired overall "look". 

Thus, in commercial applications of motion picture and television image manipulation 
processes, large amounts of time, effort and expense are expended in experimenting with the 
adjustment of operational parameters in an effort to achieve, as closely as reasonable, a desired 
result. And, this may be done for individual images, image sections, or motion scenes. 

30 One of the major applications intended for the instant invention is the incorporation of 

the algorithms disclosed herein into a film chain (a film to video transfer device). Such transfers 
are an important and costly part of the television motion picture industry. Much time and effort 
is expended in achieving desired and artistic results. And, in particular, as described above, the 
scene-by-scene color correction of such transfers is common practice. Further, such transfer 

35 from film to video can be accomplished on an off-line, frame-by-frame basis, with the 
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accompanying image processing components taking an arbitrarily large amount of time to 
accomplish the computations required for encoding a particular film frame to a particular video 
frame. 

Thus, in the instant disclosure, it will be suggested that practitioners make adjustments 
5 to the operational parameters of the disclosed algorithms in order to better achieve desired 
results. Further, it will be suggested to such practitioners that such individual adjustments may 
be applied to images or image portions exhibiting different characteristics. 

Commercial black & white and, later, color television has been available since the 1940s. 
American and Japanese systems offer 525 line frames, 30 times each second, while most 
70 European systems offer a higher resolution 625 line frame but run at a frame rate of 25 per 
second. Higher resolution military and laboratory video systems exist and, recently, a 
commercial high definition television standard (HDTV) has been developed to improve delivered 
% image quality. 7 

Lit In the US, motion picture film is projected at 48 frames per second (FPS) by showing 

%$ each of 24 pictures twice. Recently, a system was developed by Douglas Trumbull called 
fff Showscan. It provides 60 FPS, with 60 pictures each shown only once, to improve visual 
f;! quality. 

J When color was added to US black & white television, it was decided to adopt a 

D "compatible" system, which enables black & white sets to receive color television signals and 
2p display them in black & white, while color sets display the same signals in color. Similarly, it 
LI! has been suggested that the HDTV signal be compatibly receivable by standard televisions 
jrf displaying standard resolution pictures, as well as by HDTV receivers. HDTV provides both 
more video lines and more pixels (from Picture ELements: visual data points) per line. It has 
been suggested that the standard television channels may be used to transmit a "compatible" 
25 standard resolution signal while a second channel (not receivable by a standard television) be 
used to transmit the "inbetween" higher resolution information. However, HDTV may also 
display a wider picture when compared with standard television. Inclusion of the extra "side 
strips" in a compatible broadcast system has been one of the main problems. 

It is established practice to transmit motion picture film, which has a much higher 
30 resolution and a different frame rate, over a broadcast television channel by use of a film chain. 
Essentially a motion picture projector coupled to a television camera, the film chain synchronizes 
the two imaging systems. In newer film chain systems the video camera has been replaced by 
a digital image sensor and digital frame store. In the US, each video frame consists of two 
interleaved video fields, resulting in 60 fields per second. US film runs at 24 frames per second. 
35 This results in a ratio of 2.5 video fields per film frame. Practically, this is achieved by 
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alternating 3 repeated video fields and 2 repeated video fields for alternate film frames. The 
spatial resolution of the image is reduced by the characteristics of the video camera. 

It is also established practice to generate synthetic television signals (without a camera) 
by using electronic devices such as character (text) generators, computer graphic systems and 
5 special effects generators. 

Recent developments in home televisions and VCRs include the introduction of digital 
technology, such as full-frame stores and comb filters. 

There exist many techniques for bandwidth compression of electronic signals, a number 
of which have been applied to television systems. These are particularly useful for transmitting 
70 images from space probes or for satellite transmission, where resources are limited. 

The off-line, or frame-by-frame, recording of frames computed by computer animation 
programs is a well-known standard practice. Off-the-shelf hardware and software exist that 
permit both low- and high-end graphic systems to commit such images to film or video. 8 
U! US Patent Number 4,631,584 issued to Reitmeier teaches how to display reduced 

%h resolution (but, otherwise, not encoded) side strip information incorporated into the horizontal 
01 blanking intervals of the current frame and the current line. He accomplishes this by changing 
^1 the clock signal of the electron beam in order to "stretch" (at the display) his compressed side 
J strip information. 

P The instant invention, on the other hand, utilizes vertical blanking intervals as well 

jgb horizontal and permits information (side strip or otherwise) to be stored non-adjacent to the 
|I! standard image information with which it is to associated or integrated. In particular, the 
additional information may be encoded into the blanking intervals of a frame far ahead of the 
frame containing a "center portion" with which it is to be integrated. Thus, by arriving early, 
this information may undergo a relatively long and complex process of decoding, decompression, 
25 preparation and integration before it needs to be displayed. Therefore, complex and sophisticated 
encoding mechanisms may be employed in order to compress the additional information into the 
available "empty" spaces. 



30 



35 
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BRIEF SUMMARY OF INVENTION 

The instant invention relates to a class of methods for selecting subsets of information, 
5 such as that derived by digitizing a high-definition television signal, and arranging that reduced 
amount of information into a second set of information which can then be, for example, 
converted to a low-definition signal which can then be transmitted over channels intended for 
low-definition signals, with minimal perceivable artifacts; as well as methods and devices for 
incorporating additional information, such as HDTV side strips, into the blanking intervals of 
10 a previous frame, which were developed to affect the foregoing, but which can also be 
independently practiced to great advantage. 

[if BRIEF DESCRIPTION OF DRAWINGS 

% 

m Figure 1 depicts in the top configuration a standard encoding/decoding bandwidth 

P reduction scheme. The bottom configuration depicts one class of embodiments where decoding 
f s occurs after display, as viewed, in the human eye/brain (retinex). 

CI Figure 2 depicts selection of a subset of a relatively high-resolution television signal for 

transmission over a lower bandwidth channel, where the upper left corner element (A) of each 
fjf two-by-two pixel cell, is transmitted (*) for every cell, in every frame of the source image. 
M Figure 3 depicts selection of a subset of a relatively high-resolution television signal for 

transmission over a lower bandwidth channel, where all four picture elements from each two-by- 
two cell in the source image are averaged, as by a low-pass digital filter, to derive the pixels of 
25 the destination display cell denoted by X, in every frame of the source image. 

Figures 4A through 4D each depict a point sampling scheme, similar to Figure 2; but, 
for each figure, a different pixel from the two-by-two cell is sampled, indicated by a (*). These 
four types of sampling would be alternated in a cycle of four. 

Figure 5 is a system diagram depicting the encoding of HDTV "side-strips" into NTSC 
30 blanking intervals, which can be of a previous frame. 

Figure 6 is a system diagram depicting the decoding of HDTV "side-strips" from NTSC 
blanking intervals, which can be of a previous frame. 

Figure 7 is a system diagram depicting off-line recording and real-time play-back/delivery 
of an encoded video signal. 

35 
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DESCRIPTION OF INVENTION 

The instant invention relates to a method of signal encoding that is particularly useful for 
5 preparing a high-definition television signal for transmission over a channel intended for lower 
resolution signals. It can also be used to improve the perceived quality of television systems, or 
for other purposes. 

In brief, the invention comprises a system for selecting a subset of information to be 
transmitted, from all the information in a frame, and for varying the parameters of selection 

10 from frame to frame. This results in the reduction of transmitted information, and improved 
perception on the part of the viewer when compared to non- varying selection schemes. In 
addition, techniques and systems for incorporating additional information, such as HDTV side 

P strips, into the blanking intervals of a previous frame are described, to affect the foregoing, but 

JJi which also can be independently practiced to great advantage. 

If To that end, the instant invention utilizes vertical blanking intervals as well horizontal 

% and permits information (side strip or otherwise) to be stored non-adjacent to the standard image 
f I information with which it is to associated or integrated. In particular, the additional information 
u% may be encoded into the blanking intervals of a frame far ahead of the frame containing a 
p "center portion" with which it is to be integrated. Thus, by arriving early, this information may 
§b undergo a relatively long and complex process of decoding, decompression, preparation and 
ju t integration before it needs to be displayed. Therefore, complex and sophisticated encoding 
EI mechanisms may be employed in order to compress the additional information into the available 
u - "empty" spaces. 

In order to understand the invention more fully, it is helpful to examine certain aspects 
25 of film and video display systems, their shortcomings, and the functioning of the human visual 
system. 

Theory of Operation : 

30 The basis of the instant invention is that the human visual system is capable of, and 

sometimes even works better by, reconstructing information that is presented over time. 

A simple example of this involves a picket fence. If a viewer looks fixedly through a 
picket fence, about half the scene will be hidden from view and the pickets will be clearly 
defined. However, if the viewer shifts the head back and forth, or walks briskly by the fence, 

35 the perception will be of seeing the entire scene, even though half the scene will be missing at 
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any given instant. The pickets will also become less noticeable. The human eye/brain system is 

capable of reconstructing the scene from the stream of partial views and of disregarding the 

sampling scheme imposed by the (relatively moving) pickets. 

In the parlance of this application, both these cases constitute bandwidth compression 
5 (half the scene is hidden at all times); but, the fixed view is a static STS, and the moving 

perspective view is a variable STS. Thus, the eye is capable of reconstructing information 

presented through a varying sample window. 

Other aspects of human vision indicate that varying information is preferable to static 

information. As described above, repeating identical information to the eye, as with theatrical 
10 motion picture film projection, creates the perception of muddiness or opacity. Certain structures 

in the retinex, such as edge detectors and motion detectors, only work in differential mode. That 

is, they do not work when presented with information that is temporally or spatially uniform, 
h: Another (although, longer term) phenomenon is visual fatigue or after-imaging. The receptor 
0! elements in the eye become exhausted by looking at a fixed image. The scene will eventually 
fh become almost invisible (if the view remains constant) and a negative of the scene will be 
m perceived upon finally looking away. 

£5 The human eye vibrates with a continuous tremor of 30 to 80 Hz. during its normal 

f s function. The retinex is able to reconstruct a stable detailed image from this vibrating input. It 
p may, in fact, yield a more detailed image than could be constructed from an absolutely stable 

eye and may help prevent the perception of the structure of, or defects in, the retinex itself. 9 
y I A similar situation results from the way in which motion picture film is manufactured and 

y used. Film images are made up of a random distribution of photosensitive particles collectively 
called "film grain". When blown-up on a large screen the grain structure would be more easily 
seen, however, because the distribution changes from frame to frame the grain structure is 
25 generally not perceived. There are two ways that still images are added to motion picture 
presentations. With one technique a still scene is filmed on a number of film negative frames. 
Each has a different grain structure and, thus, when the sequence is projected (or viewed on a 
video monitor) the film grain structure is not noticed. The other technique calls for "step 
printing" the same film negative on many frames of the print to be projected (or to repeatedly 
30 scan the same film frame in a film chain). In this case the same film negative frame, with the 
same grain structure, is viewed repeatedly and the grain structure becomes more noticeable. The 
frames of the motion picture print also have grain structure, however, since this structure always 
changes (unless the projector jams or is stopped) the print film grain structure is not noticed. 
The tremor function provides variation to the STS of human vision. Thus, the retinex 
35 may be better suited to deal with variable STSs than static. 
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The retinex has the ability to reconstruct a scene from bits and pieces spread over space 
and time. It is possible to take advantage of this ability by presenting a succession of moderate 
bandwidth pictures (subsets of full-definition images), taken through varying sample windows, 
and to have the retinex reconstruct a higher resolution scene. 
5 Thus, for some embodiments, the compressed transmitted data points will be directly 

displayed on a monitor of resolution that is comparable to the transmission channel. 
Reconstruction, or collection, will occur in the retinex. Improved perception will result, when 
compared with standard transmission over the same channel and display. Thus, the technique 
may be used to improve perception of material transmitted for standard systems. It is only the 
70 camera and/or transmission parts of the system that would be replaced; standard reception and 
display elements would be used in this embodiment. This is in marked contrast to most 
bandwidth compression techniques, which utilize complex coding schemes that require complex 
M decoding before display. For example, an image compressed by run-length encoding would 
ih appear as gibberish if the data were displayed directly, without proper decoding. 
|f Other embodiments will present the results on a higher definition (compared to the 

|i channel) display. The "holes" between the transmitted points may be smoothed over by 
Cl interpolative techniques, or intermediate data may be (approximately) "reconstructed" by 
f s computation. 

p Referring to Figure 1, the top configuration (100-104) depicts a standard 

Ub encoding/decoding (101/103) bandwidth reduction scheme. The bandwidth of the source (100) 
ill and display image (104) are equal and higher than the bandwidth of the channel (102). 
P The bottom configuration (110-114) depicts one class of embodiments of the instant 

™* invention. The bandwidth of the source image (110) is higher than both the channel (112) and 
the display (113) which are comparable to each other. Decoding (1 14) occurs after display (1 13), 
25 as it is viewed (114), and it is only in the retinex (114) that the image is reconstructed. 

The goal of most bandwidth compression schemes is to "squeeze" a given information 
signal into a smaller channel. A main goal of the instant invention is to "squeeze" (111) a larger 
picture into a given channel. Although this may at first sound as a semantic difference only, 
there is a significant practical difference. 
30 Broadcast television in the US is a "given channel" that is unlikely to change for the 

foreseeable future. Bandwidth compression schemes that would allow the use of a smaller 
transmission channel (102) and require complex decoding at destination (103) could not be used 
with the many millions of existing home receivers (e.g., 104). The bottom configuration (110- 
114) shown in Figure 1 requires no decoding equipment prior to display (113); decoding is 
35 accomplished in the human eye/brain (114). Therefore, it can be used with existing distribution 
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and reception/display equipment (113). 

It should also be noted that the "channel" (112) shown in the bottom part of Figure 1, 
can incorporate a standard encode-transmit-decode system (101-102-103), as shown above. In 
this way, the varying STS scheme (e.g., Ill) will function as a pre-selection or thinning 
5 function, prior to applying some other compression algorithm (e.g. 101), such as run-length 
encoding, delta encoding, etc. 

Static Sampling Schemes : 

10 The visual artifacts of film and video systems are perceivable because the STS of each 

system is constant. Many times each second, images are presented to a viewer. While these 
image streams approach viewing a real scene, they represent only a small subset of the available 
%t information. The way in which the subset of information is selected remains constant, frame 
tl! after frame after frame, while the content of each frame is different. In this way the "shape" of 
t£ the STS is re-enforced by being superimposed over many different images. Perceived visual 
m anomalies result. 

M In order to select a subset of a relatively high-resolution television signal for transmission 

I over a lower bandwidth channel, two straightforward approaches are depicted in Figure 2 and 
p Figure 3. 

W In both cases, the reduction is by a factor of four; by half in both the horizontal and 

II! vertical directions. Two-by-two cells of pixels are labeled with subscripts representing the row 
|:j and column of the cell. Within the cell, individual pixels are labeled A, B, C and D. 

In Figure 2, the upper left corner element (A) of each two-by-two pixel cell, is 

transmitted (*) for every cell, in every frame of the source image. This pixel value is used to 
25 reconstruct the entire destination cell in the destination image. Picture details that fall in pixels 

labeled B, C or D do not get transmitted or displayed at all. As a small visual detail moves 

across the screen it would pop on (as it passes A*) and off (as it passes B, C or D). 

In Figure 3, all four picture elements from each two-by-two cell in the source image are 

averaged, as by a low-pass digital filter, to derive the pixels of the destination display cell. The 
30 average of A, B, C & D is denoted by X. Here individual small details are lost as they are 

averaged with the neighboring three pixels. 

Both approaches exhibit defects. In the first, small details pop on and off, in the second, 

small detail is lost in the blur. This is because the method of selecting the subset of information 

— the STS — is constant from frame to frame. The "shape" of the static STS is re-enforced by 
35 viewing changing content upon which the same STS is repeatedly imposed. 
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Varying the STS : 

Although each is unique, the successive frames of a standard film or video presentation 
each have much redundant information in common with their neighbors. The STS of the present 
5 invention varies from frame to frame. By this scheme, the differences in the STS that is applied 
to successive frames is comparable to, or will even exceed, the differences in the content of 
successive frames. Therefore, the shape of the STS will not be re-enforced from frame to frame. 
Thus, sampling artifacts will be obscured, as they change from frame to frame, and the 
perception of visual anomalies, resulting from the overall varying sampling scheme, will be 
70 minimized. 

^ Frame Store Techniques : 

y§ For a single still image, 4:1 bandwidth compression may be achieved by the processes 

If depicted in Figure 2 or in Figure 3 for each transmitted frame. In either case, the same source 
§} image will be processed according to the same STS for each frame, resulting in a repeat of an 
fii identical destination frame. For the simple case of the instant invention, depicted in Figures 4 A 
s through 4D, 4:1 compression is also achieved but with improved perception on the part of the 
viewer. Four different destination frames are derived from the same source image, using the 
pD varying STS, and presented in sequence. (In any of the above three cases, post-transmission 

filtering, such as bi-linear interpolation, may be used to improve image smoothness). 
PI However, if it were known that a still image were to be transmitted for a number of 

frame times, the simple varying STS scheme depicted in Figures 4 A through 4D could be 
adapted to a variation that will provide further benefits. Assuming that the destination display 
25 incorporates some digital frame-store 10 or other memory function, the information from all four 
frames can be accumulated and used to completely reconstruct the original high-definition source 
image. This "accumulation mode" would not provide similar benefits if implemented with either 
of the two static STS schemes depicted in Figures 2 and 3, since the information from successive 
frames would be identical. 
30 With this scheme, once four frames were collected the oldest of the four could be 

discarded and replaced by the newest as it came in. The same principle could be applied to other 
patterns or distributions of data points and the scheme may be varied (as described below) over 
the frame or over time. 

Another technique can be implemented on a relatively high-definition display with a built- 
35 in frame store. Normally, a frame store provides data storage for each pixel that can be 
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displayed by the monitor 11 . However, by using the varying STS technique, bandwidth is 
compressed by transmitting data for only a sub-set of display points. Only some of the memory 
locations in the frame store would be required to store incoming data for an image frame. 
(Additional pixels could be used to hold data from previous frames, as in the accumulation mode 
5 described above; or, to hold intermediate data points created at the destination display.) 

By this scheme, the number of pixels (i.e. resolution) in the frame store would not need 
to be as high-definition as the display, at any given frame. However, since, over the course of 
many frames, data will be sent for all (or many) of the points that can be displayed on the high- 
resolution monitor, data storage points would be required for all those display points. This is 
10 because, in standard frame stores, a stored information point can only be displayed at one 
specific display point. Therefore, the resolution of the frame store would need to be comparable 
to that of the display, to accommodate the different points at different positions to be displayed 
over time. 

% The technique described below will allow a low-resolution frame store to be used with 

tl> a high-resolution display when implementing a variable STS scheme. The scheme described in 
S%- Figures 4A through 4D will be used as an example. For each 2x2-pixel-cell only one data point 
|!I is transmitted at each frame and is displayed at the destination position that it occupies in the 
f l source image. The other three are derived from repeated data, or intermediate data derived by 
p interpolation or reconstruction. While storage for all four pixel positions in the display cell are 
§0 needed over time, only one of four is needed at each frame. If the position of the display point 
In associated with the information storage point could be moved, then one storage space per multi- 
pi pixel cell would suffice. 

M In standard frame store devices, display data is read from digital memory and used to 

specify the color and/or intensity of pixels on the screen. The "look" of the displayed pixel is 

25 determined by the data contents of the associated memory location; the position of the displayed 
pixel is determined by the address of the memory location 12 . In some frame stores several 
different modes are available that "map" the memory locations to display screen locations in 
different ways. But for each mode the mapping is constant from frame to frame 13 . 

A specialized frame store device would facilitate the implementation of the varying STS 

30 technique. It is described below. In the specialized frame store the "look" of the pixel is still 
determined by the contents of the memory location. However, the location of the displayed pixel 
on the screen is determined by the address of the memory location plus an offset that can be 
determined in several ways. 

One way would be appropriate to the varying STS scheme described in Figures 4A 

35 thorough 4D. In this case the position of the entire raster would be adjusted to coincide with the 



- 15 - 



-Additional Information Into a Portion - Not Used For Program Content (BW.FBj 



Inventor: David M. Geshwind 



upper-left pixel of the multi-pixel cell for Figure 4A. Similarly, for other frames, shifts would 
be made to coincide with the upper-right, lower-right and lower-left for Figures 4B, 4C and 4D, 
respectively. Other patterns may also be used to "wobble" the entire display. These would be 
synchronized with a corresponding "wobble" in the video camera or transmission system. 
5 In another implementation the "wobble", perturbation or offset would be applied 

independently to individual pixels, or groups of pixels. Groups of pixels may correspond to 
small geometric areas (neighboring pixels or multi-pixel cells, for example); or a group may 
consist of a distribution of pixels spread out over a large section of, or an entire, image frame. 
In this case the complex pattern of offsets for each frame could be pre-determined and 
10 synchronized with the camera or image creation system. 

Alternately, pixel (or pixel group) position offset data could be stored along with pixel 
content data in the frame store. The display location of a pixel would then be determined by the 
O address of the memory location and an offset that was stored and associated with the individual 
fii pixel or pixel group. A specific case for 2x2 multi-pixel cells is described next. 
Us In standard frame stores pixel content data is stored for each pixel. For example, if eight 

|f* bits each were required for Red, Green and Blue display data, each pixel would require 24 bits 
p and the 2x2 cell would require 96 bits of data storage (24 bits for each of four pixels). In the 
f % specialized frame store 24 bits would be required for display content data (8 each for Red, Green 
frl and Blue). An additional 2 bits would be stored with each 24-bit pixel data. These two bits 
%b would be used, for example, as follows: 

0 EXTRA BIT 1 EXTRA BIT 2 PIXEL DISPLAY LOCATION 

0 0 Upper-left pixel of cell. 
25 0 1 Upper-right pixel of cell. 

1 0 Lower-left pixel of cell. 

1 1 Lower-right pixel of cell. 

In this way, for each 2x2-pixel cell, storage requirements have been reduced from 96 bits 
30 to 26 bits. If groups of 2x2 pixel cells were to be offset in like manner, the 2-bit position 
specification would only have to be stored once for the entire group. 

When this type of frame buffer is used with a monitor, the offset may be applied to the 
electron beam deflection mechanism 14 to implement the wobble of the entire image or 
individual pixel elements. Beam de-focussing may also be applied to cover the "holes" between 
35 transmitted pixel data. 
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Another technique may be used to fill the holes, or to other benefit. Besides adjusting the 
position of displayed pixels, the size or shape of pixels can also be adjusted. Again, these 
adjustments to the pixels may be designated by pre-determined pattern or from data stored with 
the pixel content data, and may be applied to individual pixels, pixel groups or to an entire 
5 frame uniformly. For example, one particular information point may be displayed as a single 
pixel one time, and as a 2x2 square in another frame. Or, one data point may be displayed as 
a single pixel in one frame, as a two-by-one pixel block in another frame, and as a one-by-two 
pixel block in still another frame. 

Alternately, and particularly with higher-resolution monitors, intermediate display data 
10 would be created to fill the "holes" between transmitted pixel data. This may be done using a 
digital (or analog) filter to perform simple interpolation or by a more complex reconstruction 
algorithm. The creation of the intermediate data could be done on the fly, by accessing several 
jj; data points from the frame store and computing intermediate values. But a low-cost frame store 
p] may not provide fast enough data access for complex point-oriented display processing. 
hh Therefore, several lines of frame store data may be transferred to a smaller amount of very high- 
fjl speed memory. This high speed memory can also be of higher resolution (then the rest of the 
M- frame store) so as to provide a place to temporarily store the calculated intermediate display 
I data, until they are displayed. 

Cl Multiple frame stores, of either standard or specialized construction, can be contained 

fii 

gp in a single display. They can be used to advantage if implementing a complex interpolation or 
ill reconstruction algorithm. For example, one frame store can be used to compute the incoming 

M frame while another frame store is being scanned out to display the previous frame. Two frame 

O 

stores can be alternated in this fashion, or more than two can be installed in a "pipeline" 

allowing several frame times to be used to process a frame, once it has been received. 15 
25 Several frame stores can also be used to store several sequential incoming frames. These 

can then be displayed, in interleaved manner, to implement the accumulation mode, which can 

be applied to the entire image or in limited areas. 

This discussion has made specific reference only to the sampling pattern described in 

Figures 4A through 4D. However, any of the patterns, techniques and variations described 
30 elsewhere in this disclosure can be used in conjunction with these frame store techniques. 

Variable Algorithm Over Image Frame : 

As described above, the varying STS principle may be implemented in numerous 
35 embodiments, which may be fine tuned for images or image sequences with different 
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characteristics. For example, the "accumulation mode" described above may improve visual 
quality for image sequences that have little or no movement, for a period, or for images of text 
or graphics. It is also possible that certain images or image sequences will have different 
characteristics in different areas and that the shape or position of those areas will change, over 
5 time, from frame to frame. 

Many image analysis techniques now exist and other useful techniques may continue to 
be developed. Similarly, techniques and algorithms now exist, and may later be developed, that 
can be used to encode area shapes, edges or boundaries. These algorithms and techniques can 
be used to analyze and specify various areas of different image characteristics. The analyses can 
70 be made on a single frame basis, or by comparing progression or change over two or more 
frames. 16 

Once a frame is analyzed and separated into multiple areas of different characteristics, 
?4 different versions of the encoding technique can be used for those several areas. These variations 
ill of the encoding technique can be designed so that the encoded image can be decoded using a 
\$ uniform decoding technique over the whole image. 

f|i Alternately, a multiplicity of fine tuned decoding algorithms can be used on different 

M areas of the image. The shapes of the decoding areas need not be identical to the encoding areas. 
I The different decoding areas may be determined by analysis at the destination display or, the 

p decoding areas may be derived from area specification information, to be included with the 
|jb image data in the transmitted signal or transmitted separately. 

III An image can be displayed with several decoding algorithms, even if it were encoded 

y with a single uniform algorithm. For example, although an image stream were encoded 

CI J. 

uniformly, the accumulation mode algorithm can be applied to limited areas (with little or no 
movement, or containing text or graphic information), while elsewhere, information is discarded 

25 as soon as it has been displayed. 

The method of specifying area shape can be varied depending upon: the 
encoding/decoding algorithm, image characteristics, the characteristics of camera, transmission 
or display equipment, or, constraints imposed by transmission bandwidth, computing power or 
time. For example, areas can be limited to groups of rectangles, triangles, polygons, or other 

30 shapes, or may be freely specified on any pixel, or multi-pixel cell, boundary. Limits can also 
be placed on the number of distinct areas in an image(s), or on the maximum change of area 
specification from frame to frame. 

If the constraints of image complexity, time or computing power do not allow for real- 
time analysis and/or separation of images into areas, analysis can be done off-line 17 for certain 

35 applications. For example, a relatively long time may be spent on the analysis and encoding of 
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the frames of a motion picture film, prior to transmission. The encoded results, perhaps with 
area specifications would then be stored on videotape (or other medium) for real-time playback, 
transmission, and display, at a later time. Alternately, frames can be computed, transmitted and 
recorded off-line, for real-time playback at a later time. 
5 Figure 7 depicts the "off-line recording" process described above. The dotted box in 

Figure 7 indicates which system elements one might expect to find combined in a typical 
"advanced television receiver" which are currently available at least in laboratory settings. 

For interlaced video systems, with frames consisting of two interleaved fields, each 
containing only odd or even lines, the grid can be overlaid on the whole frame, or on each field 
70 separately. 

Standard interlaced television cameras only scan half of the possible raster lines for each 
field. For some embodiments of the instant invention, it will be necessary to use cameras that 
T£ scan all lines of an interleaved frame in one field time, or can even scan at a rate higher than 
tJI one full frame for each field. One "field's worth" of information will then be selected, encoded 
W and transmitted according to the techniques described herein. For some applications, e.g. a film 
fjn chain, it will be possible to use a very high resolution camera that scans slower than real-time. 

Y± The encoded frames can then be recorded on videotape, or some other medium, for real-time 

U! 

k playback and/or transmission. 
ji6 Practical Applications : 

irf Digital technology has become commonplace in home television equipment as well as in 

broadcast equipment. Digital controls, comb filters and even whole-frame digital image stores 
are now available in home televisions and VCRs. Industrial and broadcast equipment is 

25 becoming even more sophisticated. Certain embodiments of the instant invention will prove 
useful on even the simplest of standard television equipment. Other variations will be best 
implemented using the newer digitally capable equipment. For example, the "accumulation 
mode" would require some sort of frame store. 

Since HDTV broadcast systems are not in place, standards have not been finalized, and 

30 home HDTV receivers are not yet available, it would be possible to include new hardware 
features (to implement the more complex features of the instant invention) in HDTV equipment 
without having to worry about incompatibility with existing HDTV systems. 

Regarding compatibility between HDTV and standard NTSC broadcast television: some 
variations of the present invention will provide direct compatibility by displaying the encoded 

35 low-bandwidth image directly on a standard television, while the fully decoded, processed and 
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reconstructed (or "accumulated") image would be displayed on a high definition monitor. 
However, the specific high definition system known as HDTV is wider than standard television 
by approximately 20% and the incorporation of the "side strips" is a problem not directly 
addressed above. However, there are portions of the television frame (totaling approximately 
5 10% — 15%) during the blanking intervals of the signal, unused for picture information. It is 
already common practice to store and broadcast digital information in some parts of the blanking 
intervals. 18 Utilizing the bandwidth reducing capabilities of the instant invention or other 
encoding techniques, the "side strips" could be compressed, and then encoded into the blanking 
intervals (or the blanking intervals of a previous frame, to allow more time to decode and 

10 reconstruct a more complex and dense coding scheme) and reconstructed, upon reception, by 
special HDTV televisions. The quality of the "side strips" may suffer slightly when compared 
to the rest of the image. However, those areas of the image are peripheral and therefore 

Q somewhat less critical to viewing. 

fn For those variations of the invention that are not suitable for "compatible" broadcast 

It television, or for HDTV, there are other useful applications. For example, specially encoded 
J! I "premium" channels could be distributed, by cable or direct satellite broadcast, for reception by 
CI special home televisions. Similarly, non-standard "premium" channels could be made available 
yi for delivery of special sports or entertainment events to theaters or arenas. Such premium 
q channels may require special television camera or recording equipment to produce the required 
p& signals or tapes. In particular, however, transferring movies using a special film chain would 
jit require only a limited amount of special production equipment. The popularity of pay movie 
CI channels may well justify the added production costs and even the cost of special home 
a**, receivers. 

The invention can also be used to reduce the bandwidth required to record television 
25 signals or to increase the amount or quality of information that can be recorded on existing VTR 
systems. Such systems can be incorporated into VCR (or videodisc) units to improve their 
recording and/or playback operation, or to provide extra performance when playing tapes (or 
videodiscs) that have been specially pre-recorded. Similarly, specially recorded tapes can be used 
in theaters, with special playback and/or display equipment, for high quality electronic cinema. 
30 In particular, the new digital tape recorder (DTR) can be adapted to making recordings for the 
instant invention. A version of the invention can also be used for industrial, commercial, 
broadcast or scientific image recording systems. Lastly, the varying STS scheme can be applied 
for bandwidth compression and/or improved perception when recording audio or other signals, 
as well as for image information. 
35 In addition, the mechanism of densely encoding additional information in previous 
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blanking intervals may be applied to purposes other than for the incorporation of side strip 
information. Such potential applications include the incorporation of data that may represent: 
additional definition; information that may be used to convert a monoscopic image into a left and 
right pair, or the second of a pair (see Inventor's referenced US Patent Number 4,925,294 and 
patent application number 07/951,267); various area maps which may be used to apply a 
multiplicity of algorithms to separate areas of an image; motion analysis information; interactive 
or gaming information; multimedia or hypermedia information; or, otherwise. 

The potentially complex nature of the decoding may require "pipelined architecture" at 
the receiving end in order to affect the processing of that information over several frame times 
with a multiplicity of limited processors. Further, the complex nature of the decoding, as well 
as the "look-ahead" nature of the encoding (encoded into the current frame's blanking intervals 
is information regarding a subsequent frame's information) may be facilitated by the "off-line" 
computation and recording techniques disclosed herein. 
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DESCRIPTION OF PREFERRED EMBODIMENTS 

What follows is a detailed description of drawings that will illustrate several preferred 
5 embodiments of the instant invention. 

Figure 5 depicts the encoding process (500-544) and device (520 & 530) described above 
in the section entitled "Practical Applications", for use with an HDTV signal (500 & 510) 
with "side strips" (501/502 & 511/512). Shown are two HDTV frames (500/501/502/503 & 
510/511/512/513) (which may be the same frame if M = 0) separated by M frame times. The 
70 center portion (513) of HDTV frame N (510) (roughly corresponding to an NTSC 4/3 aspect 
ratio frame) is extracted and stored or delayed (530) for M frames. The "side strips" (501 & 
502) of subsequent HDTV frame N+M (500) are encoded (520) and inserted into (544) the 
Q blanking intervals (543) of the same NTSC frame (540) into which the earlier center section of 
jlj N (513/542) is now also inserted (541) to create an NTSC frame (540) containing the "center 
is portion" of HDTV frame N (542 in 541) and the encoded "side strips" of HDTV frame N+M 
!|5 (544 in 543). 

Q Figure 6 shows the complementary process/device (600-653/640). Two frames separated 

W by M frame times (600 & 610) (which may be the same frame if M = 0) are depicted. NTSC 
pi frame N (610) contains (in 611) the "center portion" of HDTV frame N (612) and encoded in 
p& its blanking intervals (613) the "side strips" of later (by M frame times) frame N+M ( 614 in 
[J 613). Similarly, NTSC frame N+M (600) contains (in 601) the "center portion" of HDTV 
D frame N+M (602) with the encoded "side strips" of HDTV frame N+2M (604 in 603). At time 
CI N (630) the encoded information containing the "side strips" of HDTV frame N+M (614) 
(contained in the blanking intervals (613) of NTSC frame N (610)) is directed to a "slow" 
25 decoder (642) (which may take up to M frame times to decode this information back into the 
visual information representing the HDTV "side strips" 651/652)) and storage unit (642). M 
frame times later (620) the standard NTSC image (601) of frame N+M (600) which contains 
the center section of HDTV frame N+M (602), is routed by the "quick pass-through" section 
(641) to a unit which assembles (643) that information (601) with the delayed and stored "side 
30 strip" information (614) and outputs a complete HDTV frame N+M (650/651/652/653) at time 
N+M (620). 

The flows depicted in the software flow diagrams herein are exemplary, some items may 
be ordered differently, combined in a single step, skipped entirely, or accomplished in a 
different manner. However, the depicted flows will work. In particular, some of these functions 
35 may be carried out by hardware components, or by software routines residing on, or supplied 
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with, such a component. 

Similarly the systems depicted in the system diagrams herein are exemplary, some items 
may be organized differently, combined in a single element, omitted entirely, or accomplished 
in a different manner. However, the depicted systems will work. In particular, some of these 
functions may be carried out by hardware components, or by software routines residing on, or 
supplied with, such a component. 

It will thus be seen that the objects set forth above, among those made apparent from the 
preceding description, are efficiently attained and certain changes may be made in carrying out 
the above method and in the construction set forth. Accordingly, it is intended that all matter 
contained in the above description or shown in the accompanying figures shall be interpreted as 
illustrative and not in a limiting sense. 

Now that the invention has been described, what is claimed as new and desired to be 
secured by Letters Patent is: 
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CLAIMS 



1. A process for modifying an information bearing signal, said information comprising at 
least in part program content, comprising the step of: 

a. encoding into at least one portion of said information bearing signal unused for 
program content of a first segment of said information bearing signal, 
supplemental information intended to be utilized in association with, but not 
before the presentation of, a second segment of said information bearing signal. 

2. A process as in claim 1 wherein said supplemental information comprises three- 
dimensional information. 

3. A process as in claim 1 wherein said supplemental information comprises image area 
map information. 

4. A process as in claim 1 wherein said supplemental information comprises interactive 
information. 

5. A process as in claim 1 wherein said supplemental information comprises hypermedia 
information, 

6. A process as in claim 1 wherein said supplemental information comprises other than from 
the group comprising three-dimensional information, image area map information, 
interactive information and hypermedia information. 

7. A process as in claim 1 wherein said first segment and said second segment are one and 
the same. 

8. A process as in claim 2 wherein said first segment and said second segment are one and 
the same. 

9. A process as in claim 3 wherein said first segment and said second segment are one and 
the same. 
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10. A process as in claim 4 wherein said first segment and said second segment are one and 
the same. 

11. A process as in claim 5 wherein said first segment and said second segment are one and 
the same. 

12. A process as in claim 6 wherein said first segment and said second segment are one and 
the same. 

13. A process as in claim 1 wherein said first segment and said second segment are distinct, 
and said second segment is subsequent to said first segment. 

14. A process as in claim 2 wherein said first segment and said second segment are distinct, 
and said second segment is subsequent to said first segment. 

15. A process as in claim 3 wherein said first segment and said second segment are distinct, 
and said second segment is subsequent to said first segment. 

16. A process as in claim 4 wherein said first segment and said second segment are distinct, 
and said second segment is subsequent to said first segment. 

17. A process as in claim 5 wherein said first segment and said second segment are distinct, 
and said second segment is subsequent to said first segment. 

18. A process as in claim 6 wherein said first segment and said second segment are distinct, 
and said second segment is subsequent to said first segment. 

19. A process for modifying an information bearing signal, said information comprising at 
least in part program content, comprising the steps of: 

a. encoding into at least one portion of said information bearing signal unused for 
program content of a first segment of said information bearing signal, 
supplemental information intended associated with a second, distinct, subsequent 
segment of said information bearing signal; and, 

b. utilizing said supplemental information in association with said second, distinct, 
subsequent segment. 
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20. A process as in claim 19 wherein said first segment and said second segment are non- 
adjacent. 

21. A process as in claim 19 comprising in addition: 

c. said supplemental information is associated only with said second, distinct, 
subsequent segment; and, 

d. the step of processing said supplemental information is performed between steps 
a. and b. 

22. A process as in claim 20 comprising in addition: 

c. said supplemental information is associated only with said second, distinct, 
subsequent segment; and, 

d. the step of processing said supplemental information is performed between steps 
a. and b. 

23. A device for presenting an enhanced information bearing signal, which information 
comprises at least in part standard program content and at least in part supplemental 
information, comprising: 

a. a system component to derive standard program content information from a 
standard portion of said information bearing signal; 

b . a system component to derive supplemental information from information encoded 
in portions of said enhanced information bearing signal unused for standard 
program content; 

c. a system component to store said supplemental information to create stored 
supplemental presentation information; and 

d. a system component to simultaneously present said standard program content 
information and said stored supplemental information, said supplemental 
information being derived at least in part from information encoded in at least one 
previous signal segment distinct from the signal segment containing the currently 
displayed standard program content information. 

24. A product comprising the information bearing signal produced by the process of claim 
1 and recorded onto a recording medium. 

25. A product comprising the information bearing signal produced by the process of claim 
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2 and recorded onto a recording medium. 

26. A product comprising the information bearing signal produced by the process of claim 

3 and recorded onto a recording medium. 

5 

27. A product comprising the information bearing signal produced by the process of claim 

4 and recorded onto a recording medium. 

28. A product comprising the information bearing signal produced by the process of claim 
70 5 and recorded onto a recording medium. 

30. A product comprising the information bearing signal produced by the process of claim 

f si j 6 and recorded onto a recording medium. 

fk 31. A product comprising the information bearing signal produced by the process of claim 

Al 1 and distributed via a signal distribution system. 

Ijf 32. A product comprising the information bearing signal produced by the process of claim 

|L 2 and distributed via a signal distribution system. 

m 

H 33. A product comprising the information bearing signal produced by the process of claim 

%l 3 and distributed via a signal distribution system. 

34. A product comprising the information bearing signal produced by the process of claim 
25 4 and distributed via a signal distribution system. 

35. A product comprising the information bearing signal produced by the process of claim 

5 and distributed via a signal distribution system. 

30 36. A product comprising the information bearing signal produced by the process of claim 

6 and distributed via a signal distribution system. 



35 
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ABSTRACT 



5 

Methods and devices for incorporating additional information, such as: HDTV side strips; 
additional definition; stereoscopic or 3D conversion information; image area maps; motion 
analysis information; interactive or gaming information; multimedia or hypermedia information; 
into a portion of an information bearing signal unused for program content. In particular, the 

10 additional information may be placed in advance of the program material with which it is 
associated in order to permit additional time to be expended upon the decoding, preparation 
and/or use of such information by a low-powered computing element. Off-line computation and 

p recording may be used to advantage at the encoding end of the process, as well as pipelined 

f:l architecture at the reception end. 

If 



25 



30 



35 
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NOTES 

1 . Typical examples include: 

Digital Video: Selections from the SMPTE Journal and Other Publications, Society of Motion Picture and 
Television Engineers, Inc. (SMPTE), 1977. 

Digital Video Volume 2, SMPTE 1979. 

Digital Video Volume 5, SMPTE 1980. 

Graphics Engines, Margery Conner, Electronic Design News (EDN), Cahners Publishing Company, 
Newton, MA, Volume 32, Number 5, March 4, 1987, pages 112-122. 

Algorithms for Graphics and Image Processing, Theo Pavlidis, Computer Science Press 1982. 

Computer Vision, Ballard and Brown, Prentice-Hall, Englewood Cliffs 1982. 

Industrial Applications of Machine Vision, IEEE Computer Society, Los Angeles 1982. 

Structured Computer Vision, Ed. Tanimoto and Klinger, Academic Press, New York 1980. 

Computer Architecture for Pattern Analysis and Image Database Management, IEEE Computer Society 
Press, Hot Springs 1981. 

Image Processing System Architectures, Kittler & Duff, John Wiley & Sons, Inc., New York 1985. 

Multiresolution Image Processing and Analysis, Ed. A. Rosenfeld, Springer-Verlag, New York 1984. 

Image Reconstruction from Projections, Gabor T. Herman, Academic Press 1980. 

Basic Methods of Tomography and Inverse Problems, Langenberg and Sabatier, AdamHilger, Philadelphia 
1987. 

US Patent Number 2,940,005 issued June 7, 1960, Inventor: P. M. G. Toulon. 

Principles of Interactive Computer Graphics, Second Ed., Newman & Sproull, McGraw-Hill Book 
Company, New York 1979. 
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Advances in Image Processing and Pattern Recognition, Elsevier Science Publishers B.V., Amsterdam, 
1986. 

Image Recovery Theory and Application, Henry Stark, Academic Press, Inc., New York 1987. 
Handbook of Pattern Recognition and Image Processing, Ed. Tzay Y. Young, Academic Press, Inc., New 
York 1986. 

Fundamentals of Interactive Computer Graphics, Foley and Van Dam, Addison- Wesley, New York, 1982. 

Real Linear Algebra, Anatal E. Fekete, Marcel Dekker, Inc., New York 1985. 

Finite Dimensional Multilinear Algebra, Parts I & II, Marvin Marcus, Marcel Dekker, Inc., New York 
1973. 

Sparse Matrix Computations, Ed. Bunch & Rose, Academic Press, Inc., New York 1976. 

Matrix Computations and Mathematical Software, John R. Rice, McGraw-Hill Book Company, New York 
1981. 

The Architecture of Pipelined Computers, Peter M. Kogge, McGraw-Hill Book Company, New York 1981. 

Digital System Design and Microprocessors, John P. Hayes, McGraw-Hill Book Company, New York 
1984. 

Digital Filters and the Fast Fourier Transform, Ed. Bede Liu, Dowden, Hutchenson and Ross, Inc., 
Stroudsburg 1975. 

Hardware and Software Concepts in VLSI, Ed. Guy Rabbat, Van Nostrand Reinhold Company, Inc., New 
York 1983. 

Digital Signal Processing, Oppenheim and Schafer, Prentice Hall, Inc., Englewood Cliffs 1975. 
Movements of the Eyes, R. H. S. Carpenter, Pion, Limited, London 1977. 
Service Manual: DCX-3000 3-Chip CCD Video Camera, SONY Corporation. 
Color Television: Principles and Servicing 1973. 

Multi-Dimensional Sub-Band Coding: Some Theory and Algorithms, Martin Vetterli, Signal Processing 6 
(1984) 97-112, Elvsevier Science Publishers B.V. North-Holland, p. 97-112. 
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The Laplacian Pyramid as a Compact Image Code, Burt and Adelson, IEEE Transactions on 
Communications, Vol. Com-31, No. April 1983, p. 532-540. 

Exact Reconstruction Techniques for Tree-Structured Subband Coders, Smith & Barnwell, IEEE 
Transactions on Acoustics, Speech and Signal Processing, Vol. ASSP-34, No. 3 June 1986, p. 434-441. 

Theory and Design of M-Channel Maximally Decimated Quadrature Mirror Filters with Arbitrary M, 
Having the Perfect Reconstruction Property, P.P. Vaidyanathan, IEEE Transactions on Acoustics, Speech 
and Signal Processing, Vol. ASSP-35, No. 4, April 1987, p. 476-492. 

Application of Quadrature Mirror Filters to Split Band Voice Coding Schemes, Esteban & Galand, IBM 
Laboratory, 06610, La Gaude, France, 

Extended Definition Television with High Picture Quality, Broder Wendland, SMPTE Journal, October 
1983, p. 1028-1035. 

In addition, for information on the state of general knowledge in fields related to computer science, one 
may consult The McGraw-Hill Computer Science Series (and Series in Artificial Intelligence) including, but 
not limited to, the following selected titles, all published by McGraw-Hill Book Company, New York in 
1983 or before: 

The Design and Analysis of Computer Communication Networks, Ahuja 

The Design and Analysis of Instruction Set Processors, Barbacci and Siewiorek 

Computer Structures, Bell and Newell 

Digital Computer Arithmetic, Cavanagh 

Systems Programming, Donovan 

Coordinated Computing, Filman and Friedman 

Computer Organization and Programming, Grear 

Introduction to the Design and Analysis of Algorithms, Goodman and Hedetniemi 
Computer Organization, Hamacher, Vranesic, and Zaky 
Computer Architecture and Organization, Hays 
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Digital Computer Systems Principles, Hellerman 
Computer System Performance, Hellerman and Conroy 
Microprogramming Primer, Katzan 
Operating Systems, Madnick and Donovan 
Mathematical Theory of Computation, Manna 

Introduction to Simulation: Programming Techniques and Methods of Analysis, Payne 

Matrix Computations and Mathematical Software, Rice 

Software Engineering: Design, Reliability, and Management, Shooman 

Computer Structures: Principles and Examples, Siewiorek, Ball and Newell 

Introduction to Computer Organization and Data Structures, Stone 

Computing: An Introduction to Procedures and Procedure-Followers, Tonge and Feldman 

An Introduction to Computer Science: An Algorithmic Approach, Tremblay and Bunt 

Discrete Mathematical Structures with Applications to Computer Science, Tremblay and Manohar 

An Introduction to Data Structures with Applications, Tremblay and Sorenson 

Programming Languages, Tucker 

Knowledge-Based Systems in Artificial Intelligence, David and Lenat 

Computers and Thought, Feigenbaum and Feldman 

Problem Solving Methods in Artificial Intelligence, Nilsson 

Artificial Intelligence, Rich 

The Psychology of Computer Vision, Winston 
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2. See, for example: 

Digital Video: Selections from the SMPTE Journal and Other Publications, Society of Motion Picture and 
Television Engineers, Inc. (SMPTE), 1977. 

Digital Video Volume 2, SMPTE 1979. 

Digital Video Volume 3, SMPTE 1980. 

Extended Definition Television with High Picture Quality, Broder Wendland, SMPTE Journal, October 
1983, p. 1028-1035. 



3. For example: 

h\ PIP-512, PIP-1024 and PIP-EZ (software); PG-640 & PG-1280; MVP-AT & Imager-AT (software), all 

Q for the IBM-PC/ AT, from Matrox Electronic Systems, Ltd. Que., Canada. 

!f ! The Clipper Graphics Series (hardware and software), for the IBM-PC/AT, from Pixelworks, New 

Hi Hampshire. 

D TARGA (several models with software utilities) and AT- VIST A (with software available from the 

fil manufacturer and Texas Instruments, manufacturer of the TMS34010 onboard Graphics System Processor 

f:: chip), for the IBM-PC/AT, from AT&T EPICenter/Truevision, Inc., Indiana. 

p| The low-end Pepper Series and high-end Pepper Pro Series of boards (with NNIOS software, and including 

the Texas Instruments TMS34010 onboard Graphics System Processor chip) from Number Nine Computer 
Corporation, Massachusetts. 



4. For example: 

FGS-4000 and FGS-4500 high-resolution imaging systems from Broadcast Television Systems, Utah. 

911 Graphics Engine and 911 Software Library (that runs on an IBM-PC/ AT connected by an interface 
cord) from Megatek, Corporation, California. 

One/80 and One/380 frame buffers (with software from manufacturer and third parties) from Raster 
Technologies, Inc., Massachusetts. 
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Image processing systems manufactured by Pixar, Inc., California. 

And many different models of graphic-capable workstations from companies such as Apollo, SUN and 
Silicon Graphics, Inc. 



5. For Example: 

GMP VLSI Graphics Microprocessor from Xtar Electronics, Inc., Illinois. 

Advanced Graphics Chip Set (including the RBG, BPU, VCG and VSR) from National Semiconductor 
Corporation, California. 

TMS34010 Graphics System Processor (with available Software Development Board, Assembly Language 
Tools, "C" Cross-Compiler and other software) from Texas Instruments, Texas. 



6. Such as Inventor's own film colorization system subject of US Patent Number 4,606,625 issued August 
19, 1986. 



7. The Media Lab: Inventing the Future at MIT, Stewart Brand, Viking Press, New York 1987, page 72. 



8. For example: the DQ-422 VideoGraphics Interface by Diaquest, Inc., California or a similar system from 
Lyon-Lamb will permit output from IBM-PC frame buffers to be recorded single frame to 3/4" videotape 
recorders. Similarly, professional or "broadcast quality" solutions exist when using 1" or 2" high-band 
videotape recorders. 



9. Also referred to as microsaccades; see, for example: Movements of the Eyes, R. H. S. Carpenter, Pion, 
Limited, London 1977. 



10. See note 2. 
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11. See, for example: 

Fundamentals of Interactive Computer Graphics, Foley and Van Dam, Addison- Wesley, New York, 1982. 

Principles of Interactive Computer Graphics, Second Ed., Newman & Sproull, McGraw-Hill Book 
Company, New York 1979. 

12. See note 13. 

13. See note 3. 

14. See, for example: Color Television: Principles and Servicing 1973; and, Note 13. 

15. The Architecture of Pipelined Computers, Peter M. Kogge, McGraw-Hill Book Company, New York 1981 . 

16. See for example: 

Algorithms for Graphics and Image Processing, Theo Pavlidis, Computer Science Press 1982. 

Computer Vision, Ballard and Brown, Prentice-Hall, Englewood Cliffs 1982. 

Industrial Applications of Machine Vision, IEEE Computer Society, Los Angeles 1982. 

Structured Computer Vision, Ed. Tanimoto and Klinger, Academic Press, New York 1980. 

Multiresolution Image Processing and Analysis, Ed. A. Rosenfeld, Springer- Verlag, New York 1984. 

Advances in Image Processing and Pattern Recognition, Elsevier Science Publishers B.V., Amsterdam, 
1986. 

Handbook of Pattern Recognition and Image Processing, Ed. Tzay Y. Young, Academic Press, Inc., New 
York 1986. 

Digital Signal Processing, Oppenheim and Schafer, Prentice Hall, Inc., Englewood Cliffs 1975. 
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17. For example: the DQ-422 VideoGraphics Interface by Diaquest, Inc., California or a similar system from 
Lyon-Lamb will permit output from IBM-PC frame buffers to be recorded single frame to 3/4" videotape 
recorders. Similarly, professional or "broadcast quality" solutions exist when using 1" or 2" high-band 
videotape recorders. 



18. For example, SMPTE time code signals, "user bits", and certain color reference signals are sometimes 
present during the vertical blanking signal. 
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